Skip to content

feat(deepnote): one notebook per .deepnote file#429

Draft
tkislan wants to merge 28 commits into
mainfrom
tk/single-notebook
Draft

feat(deepnote): one notebook per .deepnote file#429
tkislan wants to merge 28 commits into
mainfrom
tk/single-notebook

Conversation

@tkislan

@tkislan tkislan commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

@coderabbitai ignore

Migrates the extension to a single notebook per .deepnote file model, removing the multi-notebook-per-file machinery (the ?notebook=<id> URI selection and its timers/manager state).

What changes

  • Deserialize renders one notebook — the first non-init notebook (falling back to the init only when it is the file's only notebook); serialize resolves the target from document metadata with an exact (projectId, notebookId) lookup.
  • Legacy multi-notebook files are split on demand — opening one shows a notification that writes one new single-notebook file per notebook, migrates the environment selection, then deletes the original (write-before-delete; the original is never lost on failure).
  • New / duplicated notebooks become sibling files (never appended into one file).
  • Project metadata fans out across siblings — integration/rename edits are written to every sibling .deepnote of the project on disk (open or closed), with file-watcher self-write suppression.
  • Explorer is grouped by project (ProjectGroup → file → notebook) plus a status bar showing the active notebook with a "Copy details" command.
  • Server & environment are keyed per notebook (notebook.uri.toString(), 1:1 with the kernel/controller). Environment deletion now actually stops every server using it (including closed-but-running ones), fixing a real bug.
  • Snapshots are notebook-scoped with a backward-compatible reader (legacy project-scoped snapshots still load as a fallback and are never migrated) and a deferred, output-settled save.
  • Init notebook runs per kernel from its sibling file — re-running after an in-place restart (tracked in a WeakSet<IKernel>, not a persistent project flag).

How it was built

Implemented in 7 sequential, independently-reviewed chunks (8 commits). Each chunk was implemented, reviewed against the plan, unit-tested against the plan's use cases, and committed green.

Verification

  • tsc clean, both esbuild bundles build, prettier --check clean.
  • Unit suite: 2501 passing, 234 pending.
  • The 1 failing TextBlockConverter test and 3 cloud-sql tsc errors are pre-existing local symlink drift (the @deepnote/* packages are ahead of the repo locally; CI uses the published versions) — not introduced by this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro

tkislan and others added 8 commits June 23, 2026 22:23
…d) + add project-id resolver

Chunk 1 of single-notebook migration (§4 partial, §5). No behaviour change.

- Manager caches originals in a nested Map<projectId, Map<notebookId, project>>
  so sibling files sharing a project.id no longer clobber each other.
- New API: getOriginalProject(projectId, notebookId) exact/no-fallback,
  getAnyProjectEntry(projectId), storeOriginalProject/updateOriginalProject
  (3-arg), updateProjectIntegrations iterates all entries.
- Update IDeepnoteNotebookManager and IPlatformDeepnoteNotebookManager; repoint
  all project-level read-only callers to getAnyProjectEntry.
- Add canonical readDeepnoteProjectFile and resolveProjectIdFor{File,Notebook}.
- Selection state and init-run tracking intentionally kept (removed in later chunks).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…rop selection machinery

Chunk 2 of single-notebook migration (§1 + Cleanup).

- deserializeNotebook renders the first non-init notebook (findDefaultNotebook),
  falling back to the only/init notebook; never composes init.
- serializeNotebook resolves the target from document metadata alone (projectId +
  notebookId required) and looks it up with the exact getOriginalProject, throwing
  clear errors instead of falling back to a wrong sibling.
- detectContentChanges collapses to a single-notebook comparison.
- Remove the ?notebook=<id> selection machinery: findCurrentNotebookId, the
  manager's selection state + interface methods, the explorer's query-param opens
  and selectNotebookForProject calls, and the tree item's custom resourceUri.
- Explorer no longer depends on IDeepnoteNotebookManager.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…k siblings

Chunk 3 of single-notebook migration (§0, §2, §3).

- Add allocateSiblingUri: the single filesystem-aware, collision-safe sibling
  filename allocator (bumps -2/-3 before .deepnote, honors an in-batch reserved
  set, bounded retries).
- Add a notebook file factory (buildSingleNotebookFile / buildSiblingNotebookFileUri)
  for creating sibling single-notebook files (wired into the explorer in a later
  chunk).
- Add DeepnoteMultiNotebookSplitter: on opening a multi-notebook .deepnote file,
  offer to split it into one new single-notebook file per notebook. The action
  flushes the editor if dirty, writes all children, migrates the environment
  selection, then closes the tab and deletes the original to trash. A child-write
  failure leaves the original intact (write-before-delete).
- Wire the splitter into activation with an optional (desktop-only) environment
  mapper; add a refresh() passthrough on the explorer.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
Chunk 4 of single-notebook migration (§6).

- Add DeepnoteProjectMetadataPropagator (desktop): given a project id and a
  project-level mutator, enumerate every sibling .deepnote file on disk (open or
  closed), apply the change, and write it back. Skips no-op writes, refreshes the
  manager cache for open siblings, and collects per-file failures instead of
  aborting. Fires an onFileWritten hook so the file watcher treats each write as a
  self-write (no reload/save storm).
- Route integration updates and project rename through the propagator so closed
  siblings stay consistent; web falls back to the cache-only / single-file paths.
- Expose getOriginalProject/updateOriginalProject on the platform manager interface.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…us bar

Chunk 5 of single-notebook migration (§7).

- Tree is grouped: ProjectGroup (by project id) -> ProjectFile -> Notebook. A
  single-notebook file is a leaf labelled with its notebook; legacy multi-notebook
  files stay collapsible. The init notebook is excluded from counts everywhere.
- Refresh is grouping-safe: refreshNotebook evicts every sibling cache entry for a
  project id and all refreshes fire a full-tree change (no per-item fires).
- Commands are project-scoped vs notebook-scoped; new/duplicate/add-notebook create
  sibling files via the factory (never appended), delete removes the file for a
  single-notebook file, and notebook names are unique within a project group.
- Add a status bar item showing the active Deepnote notebook with a
  "Copy Active Deepnote Notebook Details" command.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
Chunk 6 of single-notebook migration (§8).

- Key the server starter maps, the config handle, and the kernel auto-selector by
  notebook.uri.toString() - the same identity the kernel and controller use - so a
  notebook's server is 1:1 with its kernel. Sibling notebooks of one project no
  longer share a server; the working directory and SQL env are taken from each
  notebook's own file.
- Fix environment deletion: stop every server using the environment (including
  closed notebooks whose server is still running) before removing the mappings,
  driven from the notebook->environment mapper. Drop the dead environmentServers
  map that was never populated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…le reader

Chunk 7a of single-notebook migration (§9).

- Write snapshots with notebook-scoped filenames via @deepnote/convert
  (generateSnapshotFilename / parseSnapshotFilename), replacing the local slug and
  filename regex.
- readSnapshot resolves snapshots path-free (it runs at deserialize, which has no
  URI): glob by project id, rank the notebook-scoped match first and keep legacy
  project-scoped snapshots as a fallback, and skip an empty-output "latest" (save
  race) or a corrupt file while walking candidates. Legacy snapshots are read, never
  migrated or deleted.
- Defer the execution snapshot save until outputs settle (quiet window with a max
  wait) and cancel it on re-execute / close.
- Use convert's computeSnapshotHash on the save path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
Chunk 7b of single-notebook migration (§10).

- The init runner now subscribes to kernel start and restart events and runs the
  init notebook found in its own sibling .deepnote file (matched by project id +
  initNotebookId via isValidSiblingInitCandidate), instead of looking it up in the
  main file's notebooks.
- Track "init has run" per kernel in a WeakSet<IKernel>: a fresh kernel runs init
  once, and an in-place restart (which fires onDidRestartKernel) re-runs it so the
  kernel is re-initialized before the next user cell. A missing sibling is logged
  and skipped without permanently marking the project.
- Remove the manager's persistent init-run tracking and the selector's init
  staging; the runner owns init triggering.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0%. Comparing base (735e5f5) to head (7fba16e).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@     Coverage Diff     @@
##   main   #429   +/-   ##
===========================
===========================
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

tkislan and others added 20 commits June 24, 2026 06:54
- void the fire-and-forget onExecutionComplete call (no-floating-promises),
  matching the existing void performSnapshotSave pattern.
- Use American "behavior" in a comment.
- Add test-only technical words (basenames, initmain, Résumé, unparseable) to cspell.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…of duplicating it

The mocha ESM loader wholesale-mocked @deepnote/convert and reimplemented its pure
helpers (resolveSnapshotNotebookId, splitByNotebooks, isValidSiblingInitCandidate,
snapshot filename generate/parse, hashing, etc.). That duplicated upstream logic
with no drift detection: if convert changed, the mock silently kept the old
behavior and tests stayed green against a fiction.

- Remove the @deepnote/convert interception from build/mocha-esm-loader.js so unit
  tests exercise the real package's pure functions (and now track its actual API).
- Mock only the one genuinely side-effecting export, convertIpynbFilesToDeepnoteFile
  (real node:fs I/O), via esmock in the explorer import suites where it is used.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…r paths

From the Codex review of the PR (F1/F3/F4/F5), all independently verified:

- F1 (P1): snapshot save fetched the cached project with getAnyProjectEntry(projectId),
  which can return the wrong sibling when multiple single-notebook siblings of one
  project are open, silently skipping the snapshot write. Use the exact
  getOriginalProject(projectId, notebookId) lookup instead.
- F3: collectNotebookNamesForProject globbed **/*.deepnote without skipping snapshot
  sidecars, so stale snapshot notebook names polluted the name-uniqueness set. Filter
  snapshot files (matching the tree provider and propagator).
- F4: detectContentChanges compared notebooks[0]; for a legacy [init, main] file the
  edited notebook is not at index 0, so edits were missed and modifiedAt preserved.
  Match the notebook by id.
- F5: the deferred-save timer fired performSnapshotSave as a floating promise; wrap the
  save body in try/catch/finally so a build/write failure is logged (not an unhandled
  rejection) and execution state is always cleared.

Adds regression tests for F1 (exact lookup), F3 (snapshot exclusion), and F4 (match by id).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…el init runs

Addresses round-2 code-review findings G2 and G3 (both verified P2).

- G2: deepnoteFileChangeWatcher's snapshot block-id recovery used the project-only
  getAnyProjectEntry(projectId), which can return a different open sibling's cached
  project (siblings share project.id), leaving originalBlocks undefined and silently
  skipping recovered outputs. Use the exact getOriginalProject(projectId, notebookId)
  — the same fix already applied to snapshotService (F1), here in the watcher path
  that was missed.
- G3: moving init execution to the event-driven runner dropped the notebook-close
  cancellation that the kernel auto-selector used to provide, so closing a notebook
  mid-init left the remaining init blocks executing against a closed notebook. Tie the
  init run to a CancellationTokenSource cancelled on notebook close and dispose it in
  a finally.

Adds regression tests for both (each fails on the pre-fix code).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…ort/delete

Removes two pieces of functionality; also folds in the branch's in-progress
updates this work was layered on top of (they could not be isolated, as the
removals are interleaved with and built on top of that WIP).

Removed - project-metadata propagator:
- Delete DeepnoteProjectMetadataPropagator and its types, drop the DI binding,
  and unwire it everywhere (activation, file-change watcher self-write hook,
  integration webview, explorer rename). Project-level fields are no longer
  fanned out across sibling files: each notebook owns its own integrations, and
  project-name drift is accepted for now. Drop the now-dead updateOriginalProject
  manager method and two stale comments.

Removed - project-level explorer commands:
- Delete the exportProject and deleteProject commands (constants, command-arg
  type, registrations, package.json command defs + sidebar menus, nls titles,
  and their unit tests). Per-notebook export remains via the existing
  exportNotebook command (first non-init notebook of the file).

Also includes the branch's pending updates the above was built on: dependency
bumps (incl. @deepnote/convert 4.0), the getOriginalProject ->
getProjectForNotebook manager rename and getAnyProjectEntry removal, and
assorted snapshot/serializer/kernel adjustments.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Brings in #432 (Cloud SQL integration support). Resolved the package.json and
package-lock.json conflicts by keeping this branch's newer @deepnote/* versions
(blocks 4.6.0, convert 4.0.0, runtime-core 0.4.0); @deepnote/database-integrations
is 1.5.0 on both sides, and the Cloud SQL source from #432 merged cleanly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
@deepnote/blocks@4.6.0+ renders `text-cell-bullet` blocks with
`indent_level >= 1` using leading spaces (two per level) before the bullet
marker. stripMarkdown's bullet regex only matches at column 0, so the
leading indentation must be trimmed first for the plain-text cell value to
round-trip correctly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
…es re-exports

snapshotFiles.ts re-exported six snapshot-filename helpers from
@deepnote/convert. Remove the re-export block and import the helpers
directly from @deepnote/convert at each use site (snapshotService.ts and
the snapshotFiles unit test). snapshotFiles.ts now keeps only its local
helpers (SNAPSHOT_FILE_SUFFIX, isSnapshotFile, extractProjectIdFromSnapshotUri)
plus the single internal use of parseSnapshotFilename.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Refactor the buildSnapshotPath method to accept an object as an argument, improving readability and maintainability. Update all relevant calls to this method throughout the snapshotService and its unit tests to match the new signature. This change enhances the clarity of parameter usage and reduces the risk of errors when passing arguments.
Trim the single-notebook test suites by removing duplicate and
tautological tests and collapsing/merging several others, shrinking the
PR's test additions by ~640 lines with no loss of real coverage.

Cuts target only tests this branch added:
- exact-(projectId, notebookId)-lookup restatements duplicated across
  the watcher, serializer, snapshot, and manager suites
- wrapper tests already covered by the delegate's own tests
  (addNotebookToProject, sibling-file allocation, project-id resolution)
- tautologies over trivial template/getter functions (serverUtils)
- framework-registration smoke tests (status bar)

Merges keep the one meaningful assertion and drop the duplicate
scaffolding (e.g. legacy-delete no-op folded into the existing delete
test; two init builders parametrized into one).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
When splitting a legacy multi-notebook .deepnote file into single-notebook
siblings, rename the original to `<name>.deepnote.legacy` instead of moving it
to the OS trash. The `.legacy` suffix takes it out of the extension's view (it
no longer matches `*.deepnote`) while keeping it on disk next to the split
results, so the user can restore it by removing the suffix.

Unlike `workspace.fs.delete({ useTrash: true })`, this is deterministic and does
not depend on an OS trash backend (which can be absent on headless Linux).
Collisions bump the name to `.legacy-2`, `.legacy-3`, … and the rename still
happens only after every child is durably written (write-before-retire).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test that drives the real VS Code UI through the
on-open split of a legacy multi-notebook .deepnote file: it asserts the split
prompt, the one-file-per-notebook result, the retained `.legacy` backup, that
each sibling opens without re-prompting, and that content plus the project
integration fan out into every split file.

Add a `createScreenshotter(this)` helper that captures step screenshots into a
per-spec directory derived from the running test file
(`test/e2e/screenshots/<spec>/`), plus the `sales-analytics.deepnote` fixture.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
…ites

Add ExTester end-to-end suites covering:
- opening a plain single-notebook file (opens directly, no split prompt, the
  status bar shows the notebook name);
- splitting a multi-notebook file that declares an init notebook (the init
  notebook becomes its own single-notebook sibling; each main sibling still
  references it via initNotebookId);
- the init-notebook runner: the sibling init notebook runs hidden in a main
  notebook's kernel so its definitions are available, and re-runs after a kernel
  restart.

Add the quick-notes and etl-pipeline fixtures (including the pre-split
extract/init siblings), and disable the kernel-restart confirmation in the E2E
settings so the restart test can drive it non-interactively.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test asserting the Deepnote Explorer groups sibling
.deepnote files by project: three files sharing one project.id collapse into a
single "Marketing" group ("3 files") whose leaves are the three notebooks, while
a file from a different project appears as its own group. Reads the tree by
diffing visible leaves before/after expanding (avoids the page-object library's
flaky CustomTreeItem.getChildItems). Adds the three marketing fixtures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
When several E2E suites run in one ExTester session (as in CI, via the
`*.e2e.test.js` glob), every workspace-folder open after the first failed with
"Failed to open folder after 5 attempts" in the suite's `before all` hook — only
the alphabetically-first suite passed.

Root cause: the simple "Open Folder" dialog (files.simpleDialog.enable)
navigates one directory level *toward* the typed path per OK click and only
accepts the folder once the browser is AT it. The helper clicked OK once then
re-opened the dialog each attempt, which reset navigation back to the default
directory — for the 2nd+ open that default is the previous, now-deleted
workspace, so the dialog fell back to "/" and never converged on the target.

Fix: click OK repeatedly within a single dialog until the pre-open workbench
element detaches (reload = folder accepted), instead of re-opening per attempt;
and set `window.openFoldersInNewWindow: "off"` so "Open Folder" reuses the
current window, keeping that reload detectable. Verified with four suites (16
tests) opening four folders in one session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test for the Deepnote status-bar item: it shows the
active notebook's name (with the "Copy Active Deepnote Notebook Details"
tooltip), hides when a non-notebook editor is focused, and — on click — copies
the notebook details to the clipboard with a confirmation toast. The clipboard
is verified by pasting into a scratch text file and reading it back.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test for the notebook-management commands that create
and rename sibling .deepnote files from the Deepnote explorer: New Notebook,
Add Notebook (project-group context menu), Duplicate Notebook, and Rename
Notebook — each verified by the resulting notebook name inside the sibling files
plus the confirmation toast. Delete Notebook is included as a pending test: its
context-menu -> native confirmation-modal interaction is unreliable to drive
under ExTester (documented inline), so it is left as a manual check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test for the Deepnote integrations UI: opening
"Manage Integrations" for a notebook whose project declares an integration lists
it (the "Sales BigQuery" integration on the sales-analytics-revenue fixture),
while a plain notebook (quick-notes) shows no such integration. Adds the
sales-analytics-revenue fixture (a single-notebook split of the Sales Analytics
project carrying the BigQuery integration + its SQL cell).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant